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Over  the  course  of  this  research  contract,  considerable  progress  was  made  in  all  the  areas  discussed 
in  the  proposal,  namely  tensor  methods  for  nonlinear  equations  and  unconstrained  optimization,  local 
strategies  for  noniinearly  constrained  optimization,  globally  convergent  strategics  for  nonlinearly  con¬ 
strained  optimization,  and  generalized  nonlinear  least  squares  (onhogonal  distance  regression).  We  sum¬ 
marize  the  work  in  those  areas  in  Sections  1-4.  In  addition,  we  have  worked  on  several  other  topics, 
namely  parallel  methods  for  optimization,  and  several  topics  related  to  unconstrained  optimization.  This 
work  is  summarized  in  Sections  5  and  6.  Section  7  contains  a  listing  of  publications  and  reports  sup¬ 
ported  by  this  contract,  and  Section  8  contains  a  list  of  research  personnel  supported  by  this  contract 

1.  Tensor  Methods  for  Nonlinear  Equations  and  Unconstrained  Optimization. 

Under  our  previous  research  contract  we  had  developed  a  new  class  of  methods,  called  tensor 
methods,  for  solving  systems  of  nonlinear  equations.  This  research  produced  very  robust  and  efficient 
general  purpose  methods  that  arc  especially  effective  for  solving  singular  problems.  Tensor  methods 
were  first  developed  for  solving  nonlinear  equations  problems  where  the  Jacobian  is  available  at  each 
iteration,  as  in  Newton’s  method.  This  method  augments  the  standard  linear  model,  the  first  two  terms  of 
the  Taylor  series,  by  a  low  rank,  second  order  tensor  term  which  is  chosen  to  allow  the  model  to  interpo¬ 
late  some  function  values  at  previous  iterates.  This  is  done  is  such  a  way  that  the  cost  of  forming,  storing, 
and  solving  the  model  is  not  significantly  higher  than  for  Newton’s  method.  In  extensive  computational 
tests,  this  tensor  method  was  considerably  more  efficient  than  a  standard  algorithm  based  on  Newton’s 
method,  with  average  gains  of  about  40%  on  the  harder  test  problems  where  F’(x*)  is  nonsingular,  and 
gains  of  around  60%  and  50%  on  the  harder  problems  where  rank  F’(x*)  =  n- 1  or  n-2  respectively.  This 
work  is  described  in  a  paper  by  Schnabel  and  Frank  published  in  the  SIAM  Journal  on  Numerical 
Analysis,  and  forms  a  portion  of  Frank’s  Ph.D  thesis. 

We  also  developed  a  tensor  method  for  nonlinear  equations  for  situations  where,  as  in  Broyden’s 
method,  Jacobians  are  unavailable  and  the  algorithm  is  based  solely  on  function  values  at  the  iterates.  It 
is  more  difficult  to  make  effective  use  of  a  second  order  tenn  in  this  case,  but  this  tensor  method  still  pro¬ 
duced  average  gains  of  about  10%  on  nonsingular  problems,  and  about  25%  and  33%  when  rank  F’(x*)  = 
n-1  or  n-2,  respectively.  The  secant  tensor  method  is  described  in  Frank’s  Ph.D  thesis,  and  in  a  paper  by 
Schnabel  and  Frank  in  The  State  of  the  Art  in  Numerical  Analysis,  M.J.D.  Powell  and  A.  Iserles,  eds. 

Our  main  accomplishment  in  tensor  method  research  under  this  contraa  has  been  to  successfully 
extend  tensor  methods  to  unconstrained  optimization  problems.  This  research  has  been  conducted  with  a 
Ph,D  student,  Ta-Tung  Chow,  who  has  been  supported  by  this  contract.  The  extension  of  tensor  methods 
to  unconstrained  optimization  is  not  straightforward.  We  have  first  considered  the  case  when  the  Hessian 
matrix  is  avaiinbi<»,  and  have  used  a  fourth  order  tensor  model,  rather  than  a  third  order  model  which 
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would  be  the  analog  of  the  nonlinear  equations  case.  The  reasons  for  using  a  fourth  order  model  are  that  a 
third  order  model  will  be  unbounded  below,  and  that  fourth  order  information  is  needed  to  accelerate  con¬ 
vergence  on  singular  problems.  We  form  this  model  by  interpolating  the  function  and  gradient  values  at 
previous  iterates;  in  practice  it  seems  that  it  is  only  effective  to  interpolate  the  information  from  the  most 
recent  past  iterate.  The  additional  cost  of  forming,  storing,  and  solving  this  model  is  not  significant  in 
comparison  to  the  standard  cost  of  Newton’s  method.  In  our  experiments,  the  tensor  method  is  consider¬ 
ably  more  efficient  than  a  comparable  method  based  on  quadratic  models,  with  average  savings  of  at  least 
30%  on  both  tK>n-singular  and  singular  problems.  This  research  has  been  presented  at  several  confer¬ 
ences,  and  in  Chow’s  M.S.  thesis,  and  is  currently  being  prepared  for  publication. 

Recently  we  have  begun  work  on  applying  tensor  methods  for  unconstrained  optimization  to  secant 
methods,  where  the  Hessian  matrix  is  approximated  from  gradient  (and  function)  values.  This  will  be  one 
of  our  main  projects  under  our  new  ARO  contract  Both  tensor  method  projects  will  be  included  in 
Chow’s  Ph.I5  thesis. 

We  have  also  continued  to  develop  tensor  methods  for  nonlinear  equations  along  with  another  gra¬ 
duate  student,  Ali  Bouaricha.  We  have  extended  the  method  to  nonlinear  least  squares  problems,  and 
developed  a  high  quality  piece  of  software  that  solves  both  nonlinear  equations  and  nonlinear  least 
squares  problems  by  tensor  methods.  This  code  also  uses  a  trust  region  method  where  earlier  research 
used  a  line  search.  This  research  is  described  in  Bouaricha’s  M.S.  thesis,  and  is  being  prepared  for  publi¬ 
cation. 

2.  Local  strategies  for  nonlineariy  constrained  optimization. 


/ 

s 


1 


Most  of  our  research  on  local  strategies  has  involved  reduced  Hessian  methods  for  constrained 
optimization.  By  this,  we  refer  to  successive  quadratic  programming  methods  where  only  the  Hessian  of 
the  Lagrangian  with  respect  to  the  null  space  of  the  constraints  is  approximated.  We  have  made  a  serious 
study  of  the  use  of  a  second  order  correction  step  of  the  type  proposed  by  Roger  Fletcher  in  conjunction 
with  reduced  Hessian  methods.  A  paper  by  Richard  Byrd  showing  that  such  a  correction  results  in  a 
better  asymptotic  convergence  rate  has  appeared  in  SIAM  Journal  on  Numerical  Analysis.  This  paper 
also  shows  that  a  similar  improvement  can  occur  with  standard  SQP  methods.  We  have  also  done  com¬ 
putational  testing  on  the  effect  of  a  second  order  correction  step  with  several  constrained  strategies. 

A  problematic  issue  in  using  the  reduced  Hessian  is  the  choice  of  basis  for  the  null  space.  We  have 
mathematically  investigated  the  issue  of  continuity  of  the  null  space  basis  and  shown  that  such  continuity 
is  impossible  to  attain  globally  in  most  cases.  We  have  also  devised  an  algorithm  that  avoids  the  discon¬ 
tinuity  difficulties  by  making  the  reduced  Hessian  method  independent  of  basis  choice.  A  paper  describ¬ 
ing  this  woiic,  by  Richard  Byrd  and  Robert  Schnabel,  has  appeared  in  Mathematical  Programming. 

Although  we  and  others  have  made  progress  on  secant  methods  for  nonlineariy  constrained  optimi¬ 
zation,  there  is  a  lack  of  understanding  of  the  behavior  of  these  methods  except  very  close  to  the  solution. 
Therefore  we  have  decided  to  try  to  extend  the  somewhat  global  results  of  Powell  on  convergence  of  the 
BFGS  method  with  arbitrary  starting  Hessian  in  the  convex  case  to  some  constrained  methods.  Our  first 
achievement  along  these  lines  is  to  extend  Powell’s  result  for  the  BFGS  method  to  the  restricted  Broyden 
class  except  for  the  DFP.  This  work  has  been  described  in  a  paper  by  Richard  Byrd,  Jorge  Nocedal  and 
Ya-Xiang  Yuan  published  in  SIAM  Journal  on  Numerical  Analysis. 
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3.  Globally  convergent  strategies  for  nonlinearly  constrained  optimization. 

Work  in  this  area  has  concentrated  on  trust  region  methods  for  nonlinearly  equality  constrained 
problems.  In  unconstrained  optimization  trust  regions  have  proven  to  be  effective  devices  for  guarantee¬ 
ing  convergence  to  the  solution  of  optimization  problems  from  poor  starting  points.  We  have  devised  an 
algorithm  which  uses  trust  regions  in  conjunction  with  successive  quadratic  programming  for  which  we 
can  prove  the  standard  global  convergence  results  without  any  assumption  of  positive  definiteness  of  the 
Hessian  approximations.  In  addition,  we  can  guarantee  that  the  limit  point  satisfies  second  order  optimal¬ 
ity  conditions  (the  first  time  this  has  been  shown  for  a  nonlinearly  constrained  optimization  algorithm), 
and  that  no  Maratos  type  effect  can  cause  the  trust  region  to  interfere  with  rapid  local  convergence. 
Unfortunately  the  method  is  stiU  sensitive  to  near  linear  dependence  of  the  constraint  gradients.  This  is  a 
serious  defect,  which  is  shared  by  almost  all  methods  for  nonlinearly  constrained  optimization.  These 
theoretical  results  have  been  presented  in  a  paper  by  Richard  Byrd,  Robert  Schnabel,  and  Gerald  Shultz  of 
Metropolitan  State  College  in  Denver  in  SIAM  Journal  on  Numerical  Analysis.  The  method  has  been 
implemented  with  the  aid  of  Emmanuel  Omojokun,  a  Ph.D.  student  in  this  department.  We  have  made 
several  improvements  to  the  algorithm  and  the  initial  experimental  results  are  encouraging. 

4.  Generalized  nonlinear  least  squares. 

By  generalized  nonlinear  least  squares  we  mean  a  generalization  of  standard  least  squares  in  which 
random  errors  are  assumed  to  occur  in  all  model  variables.  We  also  refer  to  this  approach  as  orthogonal 
distance  regression  (ODR).  We  have  first  considered  the  case  of  this  problem  where  the  model  is  given 
by  an  explicit  formula.  Although  the  number  of  variables  in  the  problem  is  proportional  to  the  number  of 
data  points,  we  have  devised  a  Levenberg-Marquardt  type  algorithm  in  which  the  work  per  step  is  only 
linear  in  the  number  of  data  points.  A  program  implementing  this  algorithm  has  been  used  on  several 
fitting  problems  arising  firom  engineering  applications,  and  an  analysis  has  been  done  of  its  convergence. 
A  paper  by  Paul  Boggs  of  NBS,  Richard  Byrd,  and  Robert  Schnabel  describing  and  analyzing  our  algo¬ 
rithm  and  giving  test  results  has  appeared  in  SIAM  Journal  on  Scientific  and  Statistical  Computing.  In 
addition  a  paper  on  the  algorithm  has  appeared  in  the  proceedings  of  the  Hghteenth  Symposium  on  the 
Interface  of  Computer  Science  and  Statistics.  With  the  help  of  Janet  Donaldson  of  NBS  the  algorithm  has 
been  implemented  in  a  good  quality  software  package,  ODRPACK,  and  a  paper  by  Paul  Boggs,  Richard 
Byrd,  Janet  Donaldson  and  Robert  Schnabel  has  been  submitted  to  ACM  Transactions  on  Mathematical 
Software.  This  package  has  already  attracted  over  20  users  around  the  country  and  the  world. 

As  a  byproduct  of  analyzing  this  algorithm  we  have  been  able  to  derive  some  new  results  on  local 
behavior  of  trust  region  algorithms.  Specifically  we  have  been  able  to  show  that  a  standard  trust  region 
strategy  using  a  bounded  Hessian  approximation  converges  linearly  if  it  converges.  This  implies  linear 
convergence  of  the  More  implementation  of  the  Levenberg-Marquardt  algorithm,  for  example.  We  are 
currently  writing  up  this  research. 

We  have  also  completed  a  statistical  comparison  of  our  ODR  method  to  standard  nonlinear  least 
squares  on  a  number  of  models,  in  cases  when  there  is  experimental  error  in  all  model  variables.  The 
study  indicated  that  the  estimates  produced  by  ODR  were  as  good  or  better  than  those  produced  by  stan¬ 
dard  nonlinear  least  squares  in  all  the  cases  we  tested.  A  paper  on  this  research  bv  Paul  Boggs,  Janci 
Donaldson,  Robert  Schnabel,  and  Qifford  Spiegelman  of  NBS  has  appeared  in  the  Journal  of 
Econometrics. 
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5.  Parallel  Methods  for  Optimization. 

During  this  research  period,  we  have  become  strongly  interested  in  parallel  computation,  especially 
parallel  methods  for  optimization.  A  general  discussion  of  this  subject  is  given  in  a  paper  by  Schnabel  in 
Computational  Mathematical  Programming,  ICSchittkowski,  ed.  One  contribution  to  parallel  optimiza¬ 
tion  during  this  period  has  been  the  construction  of  a  parallel  stochastic  global  optimization  method  and 
the  implementation  and  testing  of  this  method  on  a  network  of  Sun  workstations.  This  work  was  con¬ 
ducted  jointly  with  A.  Rinnooy  Kan  of  Erasmus  University  and  his  student  C.  Dert,  who  spent  nine 
months  at  the  University  of  Colorado.  It  is  described  in  a  paper  by  Byrd.  Dert,  Rinnooy  Kan,  and  Schna¬ 
bel  that  will  appear  in  the  proceedings  of  the  ARO  Workshop  on  Parallel  Processing  and  Medium-Scale 
Multiprocessing,  and  in  a  paper  that  has  just  been  accepted  for  publication  in  Mathematical  Program¬ 
ming. 

We  have  also  developed  parallel  methods  for  unconstrained  optimization  during  this  period.  A 
paper  by  Schnabel  in  Computer  Methods  in  Applied  Mechanics  and  Engineering  proposes  several  parallel 
methods  for  problems  with  expensive  function  evaluations  and  discusses  their  performance.  A  paper  by 
Byrd,  Schnabel,  and  Shultz  in  Annals  of  Operations  Research  motivates  cases  where  one  might  evaluate 
part,  but  not  all.  of  the  Hessian  matrix  at  each  iteration  in  parallel  optimization,  and  proposed,  analyzes, 
and  tests  several  ways  to  do  this.  Another  paper  by  Byrd,  Schnabel,  and  Shultz  that  has  been  tentatively 
accepted  for  publication  in  Mathematical  Programming  continues  the  development  of  these  methods,  and 
also  discusses  the  efficient  parallel  implementation  of  the  linear  algebraic  operations  in  secant  methods 
for  unconstrained  optimization. 

We  also  developed  some  software  tools  to  facilitate  the  use  of  a  network  of  workstations  as  a  distri¬ 
buted  memory  multiprocessor.  This  system,  called  DPUP,  is  described  in  a  paper  by  Gardner,  Gerard, 
Mowers,  Nemeth,  and  Schnabel  in  SIGNUM  Newsletter,  and  has  been  distributed  to  a  number  of  other 
sites. 

6.  Other  Research  Topics  in  Unconstrained  Optimization. 

A  project  begun  under  the  previous  research  contract  on  computational  experiments  with  trust 
region  methods  for  unconstrained  minimization  was  completed.  This  involved  computation  of  trust 
region  steps  by  minimizing  the  quadratic  model  on  a  two  dimensional  subspace  within  the  trust  region.  A 
paper  on  this  work  by  Byrd,  Schnabel  and  Gerald  Shultz  has  appeared  in  Mathematical  Programming. 

The  techniques  used  by  Byrd  and  Nocedal  in  analyzing  the  convergence  of  constrained  optimiza¬ 
tion  methods  are  of  substantial  interest  in  unconstrained  optimization  as  well.  A  paper  by  Byrd  and 
Nocedal  discussing  some  of  these  issues  has  recently  been  submitted  to  SIAM  Journal  on  Numerical 
Analysis. 

A  crucial  pan  of  most  line  search  codes  for  unconstrained  and  constrained  optimization  is  the 
modified  Cholesky  decomposition  of  Gill  and  Murray.  We  have  developed  a  new  modified  Qtolesky 
decomposition  that  has  superior  theoretical  properties  to  existing  methods,  and  appears  to  perform  better 
in  practice  as  well.  This  research  has  been  presented  at  a  conference  and  is  being  readied  for  publication. 

Finally,  John  Dennis  of  Rice  University  and  Robert  Schnabel  completed  an  extensive  survey  of 
methods  for  unconstrained  optimization,  which  will  appear  in  Handbooks  in  Operations  Research  and 
Management  Science,  Vol.  1.  Optimization. 
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